Imaging arrangements and methods therefor

ABSTRACT

Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

RELATED PATENT DOCUMENTS

This patent document claims the benefit, under 35 U.S.C. §119(e), ofU.S. Provisional Patent Application Ser. No. 60/615,179 filed on Oct. 1,2004, and of U.S. Provisional Patent Application Ser. No. 60/647,492filed on Jan. 27, 2005, both of which are fully incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates generally to imaging applications, andmore specifically to processing image data to focus and/or correct theimage data.

BACKGROUND

Imaging applications such as those involving cameras, video cameras,microscopes, telescopes and more have generally been limited in theamount of light that is collected. That is, most imaging devices do notrecord most of the information about light distribution entering thedevice. For example, conventional cameras such as digital still camerasand video cameras do not record most of the information about the lightdistribution entering from the world. In these devices, collected lightis often not amenable to manipulation for a variety of approaches, suchas for focusing at different depths (distances from the imaging device),correcting for lens aberrations or manipulating an angle of view.

For still-imaging applications, typical imaging devices capturing aparticular scene generally focus upon a subject or object in the scene,with other parts of the scene left out of focus. For video-imagingapplications, similar problems prevail, with a collection of images usedin video applications failing to capture scenes in focus.

Many imaging applications suffer from aberrations with the equipment(lenses) used to collect light. Such aberrations may include, forexample, spherical aberration, chromatic aberration, distortion,curvature of the light field, oblique astigmatism and coma. Correctionfor aberrations has typically involved the use of corrective optics,when tend to add bulk, expense and weight to imaging devices. In someapplications benefiting from small-scale optics, such as camera phonesand security cameras, the physical limitations associated with theapplications make it undesirable to include additional optics.

Difficulties associated with the above have presented challenges toimaging applications, including those involving the acquisition andaltering of digital images.

SUMMARY

The present invention is directed to overcoming the above-mentionedchallenges and others related to imaging devices and theirimplementations. The present invention is exemplified in a number ofimplementations and applications, some of which are summarized below.

According to an example embodiment of the present invention, a light isdetected with directional information characterizing the detected light.The directional information is used with the detected light to generatea virtual image, corresponding to one or both of a refocused image and acorrected image.

According to another example embodiment of the present invention, two ormore subjects at different focal depths in a scene are imaged, withportions of the scene corresponding to each subject focused at differentfocal planes. Light from the scene is focused upon a physical focalplane and detected, together with information characterizing thedirection from which the light arrived at particular locations on thephysical focal plane. For at least one subject that is located at adepth of field that is not focused upon the physical focal plane, avirtual focal plane that is different than the physical focal plane isdetermined. Using the detected light and directional characteristicsthereof, portions of the light corresponding to a focused image of theat least one subject upon the virtual focal plane are collected andadded to form a virtual focused image of the at least one subject.

According to another example embodiment of the present invention, ascene is digitally imaged. Light from the scene that is passed todifferent locations on a focal plane is detected, and the angle ofincidence of the light detected at the different locations on the focalplane is detected. A depth of field of a portion of the scene from whichthe detected light came is detected and used together with thedetermined angle of incidence to digitally re-sort the detected light.Depending upon the application, the re-sorting includes refocusingand/or correcting for lens aberrations.

The above summary of the present invention is not intended to describeeach illustrated embodiment or every implementation of the presentinvention. The figures and detailed description that follow moreparticularly exemplify these embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be more completely understood in consideration of thedetailed description of various embodiments of the invention thatfollows in connection with the accompanying drawings, in which:

FIG. 1 is a light ray capturing and processing arrangement, according toan example embodiment of the present invention;

FIG. 2 is an optical imaging device, according to another exampleembodiment of the present invention;

FIG. 3 is a process flow diagram for image processing, according toanother example embodiment of the present invention;

FIG. 4 is a process flow diagram for generating a preview image,according to another example embodiment of the present invention;

FIG. 5 is a process flow diagram for processing and compressing imagedata, according to another example embodiment of the present invention;

FIG. 6 is a process flow diagram for image synthesis, according toanother example embodiment of the present invention;

FIG. 7 is a process flow diagram for image refocusing, according toanother example embodiment of the present invention; FIG. 8 is a processflow diagram for extending the depth of field in an image, according toanother example embodiment of the present invention;

FIG. 9 is a process flow diagram for another approach to extending thedepth of field in an image, according to another example embodiment ofthe present invention;

FIG. 10 illustrates one example approach to separating light rays,according to another example embodiment of the present invention;

FIG. 11 illustrates an approach to the mapping of sensor pixel locationsto rays in the L(u,v,s,t) space with respect to data collected,according to another example embodiment of the invention;

FIG. 12 illustrates several images refocused at different depths,according to another example embodiment of the present invention;

FIG. 13A shows an imaging configuration in 2D, according to anotherexample embodiment of the present invention;

FIG. 13B shows a cone of rays from a 3D point summed for a pixel,according to another example embodiment of the present invention;

FIGS. 14A-14C show an approach to computing images with different depthsof field, according to another example embodiment of the presentinvention;

FIG. 15 illustrates an approach to the tracing of rays from a 3D pointon a virtual film plane, according to another example embodiment of thepresent invention;

FIG. 16 shows an approach to finding the value of light, according toanother example embodiment of the present invention;

FIG. 17A shows an ideal 512×512 photograph, according to another exampleembodiment of the present invention;

FIG. 17B shows an image that would be produced with an f/2 bi-convexspherical lens, according to another example embodiment of the presentinvention;

FIG. 17C shows an image computed using an image correction approach,according to another example embodiment of the present invention;

FIGS. 18A-18C illustrate the tracing of light rays common in colorimaging systems, according to another example embodiment of the presentinvention;

FIGS. 19A-19F illustrate approaches to implementing a mosaic array,according to another example embodiment of the present invention;

FIG. 20 is a process flow diagram of computational approaches torefocusing in a Fourier domain, according to another example embodimentof the present invention;

FIG. 21A shows a triangle filter approach, according to another exampleembodiment of the present invention;

FIG. 21B shows a Fourier transformation of the triangle filter approach;according to another example embodiment of the present invention;

FIG. 22 is process flow diagram of an approach to refocusing in afrequency domain, according to another example embodiment of the presentinvention;

FIG. 23 shows a set of rays which pass through a desired focal point,according to another example embodiment of the present invention;

FIGS. 24A-B show differing views of a portion of a microlens array,according to another example embodiment of the present invention;

FIG. 24C illustrates images that appear on a photosensor, according toanother example embodiment of the present invention;

FIG. 25 illustrates an example embodiment of the present invention, inwhich a virtual image is computed as it would have appeared on virtualfilm;

FIG. 26 shows an approach to manipulating a virtual lens plane,according to another example embodiment of the present invention;

FIG. 27 illustrates that the virtual film may take any shape, accordingto another example embodiment of the present invention;

FIG. 28 shows an imaging arrangement; according to another exampleembodiment of the present invention;

FIG. 29 is a process flow diagram for pre-computing a database ofweights associated with each output image pixel and each ray sensorvalue, according to another example embodiment of the present invention;

FIG. 30 is a process flow diagram for computing an output image using adatabase of weights, according to another example embodiment of thepresent invention;

FIGS. 31A-D illustrate various scalar functions which are selectivelyimplemented as a virtual aperture function, according to another exampleembodiment of the present invention;

FIG. 32 shows a virtual aperture function which varies from pixel topixel; according to another example embodiment of the present invention;

FIG. 33 is a process flow diagram of a user selecting a region of anoutput image, editing an image portion and saving the output image,according to another example embodiment of the present invention;

FIG. 34 is a process flow diagram for extending the depth of field in animage, according to another example embodiment of the present invention;and

FIG. 35 is a process flow diagram for computing a refocused image fromreceived light ray sensor data, according to another example embodimentof the present invention.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

The present invention is believed to be useful for a variety ofdifferent types of devices, and the invention has been found to beparticularly suited for electronic imaging devices and applications.While the present invention is not necessarily limited to suchapplications, various aspects of the invention may be appreciatedthrough a discussion of various examples using this context.

According to an example embodiment of the present invention, afour-dimensional (4D) light field (e.g., the light traveling along eachray in a region such as free space) is detected using an approachinvolving the determination of the amount and direction of lightarriving at a sensor located at a focal plane. The two-dimensionalposition of light in the focal plane is detected, together withinformation characterizing the direction from which the light arrived atparticular locations in the plane. With this approach, the directionallighting distribution arriving at different locations on the sensor aredetermined and used to form an image. In various discussions herein, theassembly or assemblies implemented for sensing and/or measuring of alight field are referred to as a “light ray sensor,” or a “ray sensor.”

In one application, an approach similar to the above is implementedusing an imaging system having optics and sensors that sample the spaceof light rays that are incident on an imaging plane, with computationalfunctionality that renders images from the set of measured rays indifferent ways. Each of the optics, sensors and computationalfunctionality is implemented using a variety of approaches, incombination or distinctly, depending upon the implementation. Forexample, a camera having lenses (optics) that focus an image upon aphotosensor array (sensors) located at an imaging plane can be used tosample the space of light rays. An output from the photosensor array isused with computational functions (e.g., at a processor internal and/orexternal to the camera) to render images, such as by computingphotographs that are focused at different depths or with differentdepths of field, and/or computationally correcting lens aberrations toproduce higher quality images.

In another example embodiment, optics and sensor components of animaging system direct rays of light onto sensor elements such that eachsensor element senses a set of rays including rays emanating fromspecific directions. In many applications, this set of rays is a bundleof rays that is localized in both space and direction. For manyapplications, this bundle of rays will converge to a single geometricray of light as the optics and sensor resolutions increase. In thisregard, various portions of the description herein refer to the valuessensed by the sensor elements as “rays of light” or “light rays” orsimply “rays,” even though in general they may not be limited togeometric rays.

Turning now to the figures, FIG. 28 shows an imaging arrangement 2890,according to another example embodiment of the present invention. Theimaging arrangement 2890 includes a main lens 2810, and a light raysensor that measures the value of light arriving at different locationson the sensor and from difference incident directions. In this context,measuring the value of light may be implemented by detecting lightarriving at different locations at the sensor, together withcharacteristics of that light such as intensity to generate a value.

FIG. 1 shows an imaging system 100, according to another exampleembodiment of the present invention. The imaging system 100 includes animaging arrangement 190 having a main lens 110, a microlens array 120and a photosensor array 130. In this case, the microlens array 120 andphotosensor array 130 implement a light ray sensor. Although FIG. 1illustrates a particular main lens 110 (single element) and particularmicrolens array 120, those skilled in the art will recognize that avariety of lenses and/or microlens arrays (currently available ordeveloped in the future) are selectively implemented with a similarapproach by, for example, replacing the shown main lens and/or microlensarray.

Rays of light from a single point on a subject 105 in an imaged sceneare brought to a single convergence point on the focal plane of themicrolens array 120. For instance, when the imaged point on the subject105 is at a distance from the main lens that is conjugate to thedistance from the microlens array to the main lens, the dimension “d” isabout equal to the dimension “s” as shown. A microlens 122 at thisconvergence point separates these rays of light based on the directionof the light, creating a focused image of the aperture of the main lens110 on the photosensors underneath the microlens.

The photosensor array 130 detects light incident upon it and generatesan output that is processed using one or more of a variety ofcomponents. In this application, the output light data is passed tosensor data processing circuitry 140, which uses the data together withpositional information about each photosensor providing the data ingenerating an image of the scene (e.g., including subjects 105, 106 and107). The sensor data processing circuitry 140 is implemented, forexample, with a computer or other processing circuit selectivelyimplemented in a common component (e.g., a chip) or in differentcomponents. In one implementation, a portion of the sensor dataprocessing circuitry 140 is implemented in the imaging arrangement 390,with another portion of implemented in an external computer. Using thedetected light (and, e.g., characteristics of the detected light)together with a known direction from which the light arrived at themicrolens array (as computed using a known location of eachphotosensor), the sensor data processing circuitry 140 selectivelyrefocuses and/or corrects light data in forming an image (whererefocusing may be correcting). Various approaches to processing detectedlight data are described in detail below, with and without reference toother figures. These approaches may be selectively implemented with thesensor data processing circuitry 140 consistent with the above.

Different portions of the imaging system 100 are selectively implementedin a common or separate physical arrangement, depending upon theparticular application. For example, when implemented with a variety ofapplications, the microlens array 120 and the photosensor array 130 arecombined into a common arrangement 180. In some applications, themicrolens array 120 and the photosensor array 130 are coupled togetheron a common chip or other circuit arrangement. When implemented with ahand-held device such as a camera-like device, the main lens 110,microlens array 120 and photosensor array 130 are selectively combinedinto a common imaging arrangement 190 integrated with the hand-helddevice. Furthermore, certain applications involve the implementation ofsome or all of the sensor data processing circuitry 140 in a commoncircuit arrangement with the photosensor array 130 (e.g., on a commonchip).

In some applications, the imaging arrangement 100 includes a previewarrangement 150 for presenting a preview image to a user capturingimages. The preview arrangement is communicatively coupled to receiveimage data from the photosensor array 130. A preview processor 160processes the image data to generate a preview image that is displayedon a preview screen 170. In some applications, the preview processor 160is implemented together with the image sensor 180, on a common chipand/or in a common circuit. In applications where the sensor dataprocessing circuitry 140 is implemented with the photosensor array 130as discussed above, the preview processor 160 is selectively implementedwith the sensor data processing circuitry 140, with some or all of theimage data collected by the photosensor array 130 used to generate thepreview image.

The preview image may be generated using relatively fewer computationalfunctions and/or less data than that used to generate a final image. Forinstance, when implemented with a hand-held imaging device such as acamera or cell phone, a preview image that does not effect any focusingor lens correction may be sufficient. In this regard, it may bedesirable to implement processing circuitry that is relativelyinexpensive and/or small to generate the preview image. In suchapplications, the _preview processor generates the image at a relativelylow-computational cost and/or using less data, for example by using thefirst extended depth of field computational method as described above.

The imaging system 100 is implemented in a variety of manners, dependingupon the application. For instance, while the microlens array 120 isshown with several distinguishable microlenses by way of example, thearray is generally implemented with a multitude (e.g., thousands ormillions) of microlenses. The photosensor array 130 generally includes arelatively finer pitch than the microlens array 120, with severalphotosensors for each microlens in the microlens array 120. In addition,the micolenses in the microlens array 120 and the photosensors in thephotosensor array 130 are generally positioned such that light passingvia each microlens to the photosensor array does not overlap lightpassed via adjacent microlenses.

In various applications, the main lens 110 is translated along itsoptical axis (as shown in FIG. 1, in a horizontal direction) to focus ona subject of interest at a desired depth “d” as exemplified between themain lens and an example imaging subject 105. By way of example, lightrays from a single point on the subject 105 are shown for purposes ofthis discussion. These light rays are brought to a single convergencepoint at microlens 122 on the focal plane of the microlens array 120.The microlens 122 separates these rays of light based on direction,creating a focused image of the aperture of the main lens 110 on a setof pixels 132 in the array of pixels underneath the microlens. FIG. 10illustrates one example approach to separating light rays, such that allrays emanating from a point on a main lens 1010 and arriving anywhere onthe surface of the same microlens (e.g., 1022) are directed by thatmicrolens to converge at the same point on a photosensor (e.g., 1023).This approach shown in FIG. 10 may, for example, be implemented inconnection with FIG. 1 (i.e., with the main lens 1010 implemented formain lens 110, with microlens array 1020 implemented for microlens array120, and with photosensor array 1030 implemented for photosensor array130).

The image that forms under a particular microlens in the microlens array122 dictates the directional resolution of the system for that locationon the imaging plane. In some applications, directional resolution isenhanced by facilitating sharp microlens images, with the microlensesfocused on the principal plane of the main lens. In certain applicationsthe microlenses are at least two orders of magnitude smaller than theseparation between the microlens array and the main lens 110. In theseapplications, the main lens 110 is effectively at the microlenses'optical infinity; to focus the microlenses, the photosensor array 130 islocated in a plane at the microlenses' focal depth.

The separation “s” between the main lens 110 and the microlens array 120is selected to achieve a sharp image within the depth of field of themicrolenses. In many applications, this separation is accurate to withinabout Δx_(p)·(f_(m)/Δx_(m)), where Δx_(p) is the width of a sensorpixel, f_(m) is the focal depth of the microlenses and Δx_(m) is thewidth of the microlenses. In one particular application, Δx_(p) is about9 microns, f_(m) is about 500 microns and. Δx_(m) is about 125 microns,with the separation between the microlens array 120 and the photosensorarray 130 being accurate to about 36 microns.

The microlens array 120 is implemented using one or more of a variety ofmicrolenses and arrangements thereof. In one example embodiment, a planeof microlenses with potentially spatially varying properties isimplemented as the microlens array 120. For example, the microlens arraymay include lenses that are homogeneous and/or inhomogeneous, square inextent or non-square in extent, regularly distributed or non-regularlydistributed, and in a pattern than is repeating or non-repeating, withportions that are optionally masked. The microlenses themselves may beconvex, non-convex, or have an arbitrary profile to effect a desiredphysical direction of light, and may vary in profile from microlens tomicrolens on the plane. Various distributions and lens profiles areselectively combined. These various embodiments provide samplingpatterns that are higher spatially (correspondingly lower angularly) insome regions of the array, and higher angularly (correspondingly lowerspatially) in other regions. One use of such data facilitatesinterpolation to match desired spatial and angular resolution in the 4Dspace.

FIG. 24A illustrates a view (line of sight perpendicular to the plane)of a portion of a microlens array, according to another exampleembodiment of the present invention. The microlenses are square-shapedand regularly distributed in an array.

FIG. 24B illustrates a view of a portion of a microlens array, accordingto another example embodiment of the present invention. The microlensplane distribution is not regular or repeating, and the microlenses arearbitrarily shaped.

FIG. 24C illustrates images that appear on the photosensor in connectionwith another example embodiment of the present invention, using adistribution such as that shown in FIG. 24A with a convex profile and amain lens having a circular aperture.

In other example embodiments, a regular mosaic of larger and smallermicrolenses is used. In one implementation, the resulting photosensordata is interpolated to provide a homogeneous sampling that has themaximum spatial and angular resolutions of a microlens or microlenses inthe mosaic.

FIGS. 19A-19F illustrate approaches to implementing a mosaic array suchas that described above, in connection with one or more exampleembodiments of the present invention. FIG. 19A is an overhead view of aplane showing example relative sizes and arrangement of the microlenses.FIG. 19B is an illustration of the shape of the images that form on thephotosensor array after projection through each microlens in FIG. 19A.FIG. 19C is a cross-sectional view of the array in FIG. 19A,illustrating that the microlenses have the same f-number and their focalpoints share a common plane. This requires that the smaller microlensesare positioned closer to the focal plane than the larger ones. Thiscauses the images of the main lens that appear underneath each microlensto be large without overlapping, and all appear in-focus on aphotosensor that is placed at the plane containing the focal points.

FIG. 19D shows a cross-sectional view of the microlenses shown in FIGS.19A and 19C implemented in a full imaging arrangement containing a mainlens 1910, a mosaic microlens array 1920 and photosensor arrangement1930, according to another example embodiment of the present invention.Note that although the diagram has been shown with several microlensesand several pixels per microlens, the actual number of microlenses andpixels is selected using a variety of approaches, such as by determiningthe resolution requirements of the given application and implementingappropriate numbers of each.

FIG. 19E is a Cartesian ray diagram representing the space of raysstarting at u on the main lens 1910 and terminating at s on themicrolens array 1920 (the space of rays is shown in 2D for clarity,although the full space of rays is 4D). The set of rays that are summedby each photosensor (labeled A-P) in FIG. 19C is shown on the Cartesianray diagram in FIG. 19D. In the full 4D space of rays, each photosensorintegrates a 4D box of rays. The 4D boxes for photosensors under thelarger microlenses have half the width (twice the resolution) in the (u,v) directional axes, and twice the width (half the resolution) in the(x, y) spatial axes compared to photosensors under the smallermicrolenses.

In another example embodiment, photosensor values are interpolated to aregular grid so that the resolution in all axes matches the maximumresolution in all axes. FIG. 19F illustrates such an approach, whereinthe boxes of rays that represent each photosensor value are split byinterpolating nearby box values. In the 2D ray space illustrated, eachbox is split into two, but in the 4D space, each box is split into four(split into two along each of its two longer sides). In someembodiments, the interpolated values are computed by analyzing thenearby values. In another embodiment, the interpolation is implementedas a weighted sum of the values of the original, unsplit boxes in aneighborhood of the desired value.

In some applications, the weighting is implemented in a manner thatdepends on a decision function based on the values in a neighborhood.For example, the weighting may interpolate along the axes that are leastlikely to contain an edge in the 4D function space. The likelihood of anedge near that value can be estimated from the magnitude of the gradientof the function values at those locations, as well as the components ofthe Laplacian of the function.

In another example embodiment, each of the microlenses (e.g., in thearray 1920 of FIG. 19D or similar) is tilted inwards so that theiroptical axes are all centered on the main lens aperture. This approachreduces aberrations in the images that form under microlenses towardsthe edges of the array.

Referring again to FIG. 1, the aperture sizes of the main lens 110 andof the microlenses in the microlens array 120 (e.g., the effective sizesof the opening in the lenses) are also selected to meet specificapplications in which the imaging arrangement 100 is implemented. Inmany applications, the relative aperture sizes are selected so thatcollected images are as large as possible without overlapping (i.e.,such that light does not undesirably overlap onto an adjacentphotosensor). This approach is facilitated by matching the f-numbers(focal ratios; i.e., the ratio of the aperture to the effective focallength of the lens) of the main lens and the microlenses. In thisinstance, the effective focal length, in terms of the f-number, for themain lens 110 is the ratio of the diameter of the aperture of the mainlens to the distance “s” between the main lens 110 and the microlensarray 120. In applications in which the principal plane of the main lens110 is translated relative to the plane at which the microlens array 120is located, the aperture of the main lens is selectively modified so asto maintain the ratio and thus the size of the images forming under eachmicrolens in the microlens array. In some applications, different mainlens aperture shapes such as a square aperture are used to achievedesirable (e.g., efficient) packing of the array of images under themicrolens array on the photosensor surface.

The following discussion refers to a general application of the imagingarrangement 100 of FIG. 1, in connection with one or more exampleembodiments of the present invention. Considering a two-plane lightfield “L” inside the imaging arrangement 100, L(u,v,s,t) denotes thelight traveling along a light ray that intersects the main lens 110 at(u, v) and that intersects the plane of the microlens array 120 at(s,t). Assuming ideal microlenses in the microlens array 120 and idealphotosensors (e.g., pixels) on aligned grids in the photosensor array130, all the light that passes to a photosensor also passes through itssquare parent microlens in the microlens array 120, and through thephotosensor's conjugate square on the main lens 110. These two squareregions on the main lens 110 and microlens specify a smallfour-dimensional box in the light field, and the photosensor measuresthe total amount of light in the set of rays represented by this box.Correspondingly, each photosensor detects such a four-dimensional box inthe light field; the light field detected by the photosensor array 130thus is a box-filtered, rectilinear sampling of L(u,v,s,t).

FIG. 11 illustrates an approach to the mapping of sensor pixel locationsto rays in the L(u,v,s,t) space with respect to data collected,according to another example embodiment of the invention. The approachshown in FIG. 11 and discussed herein may be applicable, for example, toFIG. 1 with each photosensor in the photosensor array 130 correspondingto a sensor pixel. Image 1170 in the bottom right is a downsampling ofraw data read from a ray sensor (photosensor) 1130, with the circularimage 1150 that forms under one microlens circled. Image 1180 in thebottom left is a close-up representation of a portion of the raw dataaround the circled microlens image 1150, with one photosensor value 1140circled within the microlens image. Since this circular image 1150 is animage of the lens aperture, the location of the selected pixel withinthe disk provides the (u, v) coordinates of the starting position of theillustrated ray 110 on the main lens. The location of the microlensimage 1150 within the sensor image 1170 provides the (x, y) coordinatesof the ray 1120.

Although the mapping of sensor elements to rays is discussed withrespect to the figures (and other example embodiments), valuesassociated with the various sensor elements are selectively representedby the value of the set of rays that is directed through optics to eachparticular sensor element. In the context of FIG. 1, each photosensor inthe photosensor array can thus be implemented to provide a value thatrepresents a set of light rays directed via the main lens 110 andmicrolens array 120 to the photosensor. That is, each photosensorgenerates an output in response to the light incident upon thephotosensor, and the position of each photosensor, relative to themicrolens array 120, is used to provide directional information aboutthe incident light.

In one example embodiment, the resolution of the microlens array 120 isselected to match a particular application's desired resolution forfinal images. The resolution of the photosensor array 130 is selected sothat each microlens covers as many photosensors as required to match thedesired directional resolution of the application, or the finestresolution of photosensors that may be implemented. In this regard, theresolution of the imaging system 100 (and other systems discussedherein) is selectively tailored to particular applications, withconsiderations such as the type of imaging, cost, complexity andavailable equipment used to arrive at a particular resolution.

Once image data is captured via optics and sensors (e.g., using imagingarrangement 190 in FIG. 1), a variety of computational functions andarrangements are implemented to selectively process the image data. Inone example embodiment of the present invention, different sets ofphotosensors capture these separated light rays from each microlens andpass information about the captured light rays to a computationalcomponent such as a processor. Images of the scene are computed from theset of measured light rays.

In the context of FIG. 1, sensor data processing circuitry 140 isimplemented to process the image data and compute images of the sceneincluding subjects 105, 106 and 107. In some applications, a previewarrangement 150 is also implemented to generate a preview image using apreview processor 160, with the preview image displayed on a previewscreen 170. The previous processor 160 is selectively implemented withthe sensor data processing circuitry 140, with a preview image generatedin a manner not inconsistent, for example, with approaches discussedbelow.

In another embodiment, for each pixel in an image output from a sensorarrangement, the computational component weights and sums a subset ofthe measured rays of light. In addition, the computational component mayanalyze and combine a set of images computed in the manner describedabove, for example, using an image compositing approach. Although thepresent invention is not necessarily limited to such applications,various aspects of the invention may be appreciated through thediscussion of several specific example embodiments of such acomputational component.

In connection with various example embodiments, image data processinginvolves refocusing at least a portion of an image being captured. Insome embodiments, an output image is generated in the context of aphotograph focused on desired elements of a particular scene. In someembodiments, the computed photograph is focused at a particular desireddepth in the world (scene), with misfocus blur increasing away from thedesired depth as in a conventional photograph. Different focal depthsare selected to focus upon different subjects in the scene.

FIG. 12 illustrates several images 1200-1240 refocused at differentdepths, computed from a single light field measured in accordance withanother example embodiment of the present invention. The approach shownin FIG. 12 may, for example, be implemented using an imaging arrangementsuch as that shown in FIG. 1.

FIGS. 13A and 13B show a refocusing approach, according to anotherexample embodiment of the present invention. This approach may beimplemented, for example, with a computational/processor component of animaging system, such as the sensor data processing circuitry 140 inFIG. 1. Each output pixel (e.g., 1301) from an imaging arrangementcorresponds to a three-dimensional (3D) point (e.g., 1302) on a virtualfilm plane 1310. This virtual film plane 1310 is located behind a mainlens 1330, where the plane 1310 is optically conjugate to the desiredfocal plane in the world (not shown). That is, the virtual film plane1310 is located at a position at which a film plane would desirably belocated to capture a simple two-dimensional (2D) image (e.g., theposition is comparable to the position at which a photographic filmwould be located with a conventional camera in order to capture a 2Dimage). By separating light by direction (e.g., using the microlensarray 120 of FIG. 1), the light arriving at the virtual film plane 1310can be selectively computed. In this regard, the value for the outputpixel 1301 is computed by summing the cone of light rays 1320 thatconverge on the corresponding 3D point 1302. Values for these rays aregleaned from the data collected by the ray sensor 1350. FIG. 13A showsan imaging configuration in 2D for visual simplicity. In FIG. 13B, acone of rays 1330 from the 3D point 1340 is summed for the same pixel1301, with the chosen world focal depth if closer to the main lens.

In some embodiments, the required light ray values do not correspondexactly to the discrete sample locations captured by the ray sensor. Insome embodiments, the light ray value is estimated as a weighted sum ofselected close sample locations. In some implementations, this weightingapproach corresponds to a four-dimensional filter kernel thatreconstructs a continuous four-dimensional light field from the discretesensor samples. In some implementations, this four-dimensional filter isimplemented with a four-dimensional tent function corresponding toquadrilinear interpolation of the 16 nearest samples in the fourdimensional space.

FIG. 35 is a process flow diagram for computing a refocused image fromreceived light ray sensor data, according to another example embodimentof the present invention. At block 3520 a set of sub-aperture images isextracted from light ray sensor data 3510, where each sub-aperture imageconsists of a single pixel under each microlens image where the pixelsare at the same relative position under its microlens. At block 3530,the set of sub-aperture images is combined to produce the final outputimage. The sub-aperture images are optionally translated relative to oneanother and composited to bring a desired plane into focus.

In another example embodiment, darkening associated with pixels near aborder of an output image is mitigated. For instance, with the pixelsnear the border of an output image, some required rays may not have beencaptured in the measured light field (they may exceed the spatial ordirectional bounds of the imaging arrangement, such as the microlensarray 120 and photosensor array 130 in FIG. 1). For applications inwhich this darkening is undesirable, the pixel values are normalized bydividing a value associated with the pixels (e.g., as captured by aphotosensor array) by the fraction of rays that are actually found inthe measured light field.

As discussed above, a variety of different computational approaches arechosen for different applications. The following discussion addressesvarious such approaches. In some applications, reference is made tofigures, and in other applications, the approaches are discussedgenerally. In each of these applications, the particular approaches maybe implemented using a computational-type component, such as the sensordata processing circuitry 140 shown in FIG. 1.

In another example embodiment, an imaging methodology for each outputpixel for a particular imaging system corresponds to a virtual cameramodel in which a virtual film is rotated or deformed, arbitrarily and/orselectively, and a virtual main lens aperture is correspondingly movedand modified in size as appropriate. By way of example, FIG. 25illustrates an example embodiment in which a virtual image is computedas it would have appeared on virtual film 2560 if it had been presentbehind a virtual lens aperture 2520 of arbitrary size on a virtual lensplane 2530 that is permitted to be non-coincident with the physical mainlens plane 2510. The value of the pixel corresponding to point 2550 onthe virtual film 2560 is computed by summing the rays passing throughthe virtual aperture 2520 and converging on point 2550, which are foundby their intersection point and incident direction on ray sensor 2570.

FIG. 26 shows an approach to manipulating a virtual lens plane,according to another example embodiment of the present invention. Avirtual lens plane 2630 and/or virtual film plant 2660 selectivelytilted, relative to a physical main lens or other reference. An imagecomputed using this approach has a resulting world focal plane that isnot parallel to the imaging plane.

In another example embodiment, as exemplified in FIG. 27, the virtualfilm 2560 need not be planar, but may take any shape.

A variety of approaches involve selective implementation of differentapertures. In one example embodiment, the virtual aperture on thevirtual lens plane is a generally circular hole, and in other exampleembodiments, the virtual aperture is generally non-circular and/or isimplemented with multiple distinct regions of any shape. In these andother embodiments, the notion of “virtual aperture” can be generalized,and in some applications, corresponds to an approach involving theprocessing of light data to correspond to light that would be receivedvia a selected “virtual” aperture.

In various embodiments, a virtual aperture approach is implemented witha pre-determined but arbitrary function on a virtual lens plane. FIGS.31A-31D illustrate various scalar functions that are selectivelyimplemented as the virtual aperture function in connection with one ormore example embodiments. The various functions include, for example,smoothly varying values (as exemplified in FIG. 31B), implementing aplurality of distinct regions (as exemplified in FIG. 31A), and takingon negative values (as exemplified in FIG. 31D). To compute the value ofa point on a virtual film, all rays that start at different points onthe virtual lens and converge on the point on the virtual film areweighted by the virtual aperture function and summed. In various otherembodiments, the final value is computed with an arbitrary computationalfunction that depends on the ray values. For example, the computationalfunction may not correspond to a weighting by a virtual aperturefunction, but may contain discontinuous program branches depending onthe value of test functions computed on the ray values.

In other example embodiments, as may be implemented in combination withthe other example embodiments described herein, a method of computingoutput pixels is chosen independently. For example, in one exampleembodiment, parameters including the orientation of a virtual lens planeand the size of the virtual aperture are varied continuously for eachoutput pixel. In another example, as illustrated in FIG. 32, the virtualaperture function used to integrate rays for each output pixel is variedfrom pixel to pixel. In an output image 3200, pixel 3201 uses virtualaperture function 3210 and pixel 3251 uses virtual aperture function3250.

In another example embodiment, a virtual aperture function varies frompixel to pixel. In one specific embodiment, the function is chosen tomask out rays from undesired portions of a particular scene, such as anundesired object in the foreground.

In another example embodiment, a human user chooses virtual apertureparameters interactively, with light data processed in accordance withthe selections. FIG. 33 is a process flow diagram showing one suchexample embodiment. In the first block 3310, the process receives datafrom a light ray sensor. In block 3320, a user selects a region of anoutput image; in block 3330 a user selects an image formation method;and in block 3340, the user alters parameters for the selected methodand visually examines a computed image of the scene at block 3350 (forexample on a computer monitor). Block 3360 checks if the user is doneediting the image portion, and if not returns to block 3330. Block 3370checks whether the user is done choosing portions of the image to edit,and if not returns to block 3320. If editing is complete, block 3380saves the final edited image.

In another example embodiment, an image with extended depth of field iscomputed by focusing on more than one subject at the same time. In oneimplementation, the depth of field of the output image is extended bysimulating conventional photographic imaging with a stopped-down(reduced size) main lens aperture. For each output pixel, an evaluationis performed using the rays of light that would have converged at theoutput pixel through an aperture (on the virtual lens plane) that issmaller than the aperture used in ray sensing.

In one implementation involving the example system 100 shown in FIG. 1,the depth of field is extended by extracting a photosensor value undereach microlens image, where each photosensor is located at the samerelative position within each microlens image. With respect to FIG. 1,extending the depth of field produces an image in which not only thesubject 105 is in focus (due to the correlation between the distances“d” and “s”) but also objects at a different depth, such as subjects 106and 107, which may otherwise be blurry due to misfocus. This approach toextending the depth of field, coupled with optional downsampling of theresulting image, is computationally efficient. This approach isselectively implemented in applications where noise generated with theimage is tolerable, such as where the image generated is for previewpurposes (e.g., for display at the preview screen 170 in FIG. 1). FIG.4, discussed below, is further directed to approaches to the generationof a preview image.

FIGS. 14A and 14B illustrate an approach to computing images withdifferent depths of field, in connection with one or more exampleembodiments. FIG. 14A shows an image and close-up computed withrefocusing. Note that the face in the close-up is blurry due to shallowdepth of field. The middle row of FIG. 14B shows the final imagecomputed with an extended depth of field approach such as that describedin the previous paragraph.

FIG. 8 is a process flow diagram for another computational approach toextending the depth of field in an image, according to another exampleembodiment of the present invention. At block 810, a set of imagesrefocused at all focal depths in a particular scene is refocused. Atblock 820 and for each pixel, a pixel is determined from a set of imagesthat has the highest local contrast. At block 830, the pixels having thehighest local contrast are assembled into a final virtual image. Withthis approach, a desirable signal-to-noise ratio (SNR) can be obtained,using a relatively high number of pixels (e.g., relative to selecting asingle pixel (photosensor) for each microlens in a microlens array).Referring to FIG. 14C, the example image shown is produced using anapproach similar to that described in connection with FIG. 8 andexhibits relatively low image noise.

In one alternative embodiment, a minimum set of refocused images tocompute is defined as follows, in terms of the distance between avirtual film plane for each refocused image and the principal plane ofthe main lens via which the light for the image is passed to the virtualfilm plane. A minimum distance is set at the focal length of the mainlens, and the maximum distance set at the conjugate depth for theclosest object in the scene. The separation between each virtual filmplane is no more than Δx_(m)f/A, where Δx_(m) is the width of amicrolens, f is the separation between the main lens and the microlensarray, and A is the width of the lens aperture.

In another example embodiment, refocused images are combined to producean extended depth of field image at each final pixel to retain the pixelthat is best focused in any of the set of refocused images. In anotherembodiment pixels to retain are chosen by enhancing the local contrastand coherence with neighboring pixels. For general information regardingimaging, and for specific information regarding approaches to imaginginvolving enhancing local contrast, reference may be made to Agarwala,A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B.,Salesin, D., Cohen, M., Interactive Digital Photomontage, in ACMTransactions on Graphics, 23, 3 (2004), 292-300, which is fullyincorporated herein by reference.

In another example embodiment of the present invention, an extendeddepth of field image is computed as follows. For each output imagepixel, a refocusing computation is performed at the pixel to focus atdifferent depths. At each depth, a measure of the homogeneity of therays that converge is computed. The depth that produces the (relative)maximum homogeneity is chosen and kept for that pixel value. With thisapproach, where an image pixel is in focus, all of its rays originatefrom the same point in the scene and thus are likely to have similarcolor and intensity.

Although the measure of homogeneity can be defined in various ways, formany applications, the following measure of homogeneity is used: foreach color component of each ray, the squared difference of that colorintensity is computed from the corresponding color component of thecentral ray (the ray that arrives at the pixel at an angle closest tothe optical axis of the main lens). All of these squared differences aresummed, and the homogeneity is taken to be the reciprocal of the sum.

FIG. 34 is a process flow diagram for extending the depth of field in animage, according to another example embodiment of the present invention.At block 3410, a pixel is selected in a virtual image to be computed. Atblock 3420, the pixel is refocused at a plurality of focal depths, andthe homogeneity of the rays that are combined to refocus at each depthis computed. At block 3430, the refocused pixel value associated withthe highest homogeneity of rays that are combined is retained as thefinal output image pixel value. The process continues at block 3440until all pixels are processed.

In another example embodiment, the above process is adapted so that theselection of final pixel values takes into account the neighboring pixelvalues, and the homogeneity of the associated rays that are combined tocompute those pixel values.

In other example embodiments of the present invention, the depth offield is extended by focusing each pixel on the depth of the closestobject in that direction. FIG. 9 is a process flow diagram for extendingthe depth of field in an image, according to one such exampleembodiment. At block 910, a pixel is selected in a final virtual imageto be computed. At block 920, the depth of a nearest object is estimatedfor a light ray (or set of light rays) traced from the selected pixelinto a scene through the center of the lens.

At block 930, the selected pixel's value is computed in an imagerefocused at the estimated depth. If additional pixels are desirablyprocessed at block 940, another pixel is selected at block 910 and theprocess continues at block 920 for the newly-selected pixel. When noadditional pixels are desirably processed at block 940, the computedvalues for each selected pixel are used to create the final virtualimage.

In some embodiments involving the extension of the depth of field, avalue at each output pixel is computed by disregarding light rays thatoriginate at depths closer than the depth of a desired object tomitigate or eliminate artifacts such as those generally referred to as“blooming” or “halo” artifacts around the borders of objects closer tothe lens. By way of example, FIG. 23 illustrates the set of rays thatpass through a desired focal point 2301 in the world on a subject ofinterest 2310. Some of these rays are occluded from the main lens byobject 2320, and these correspond to the light rays 2340 that aredetected by ray sensor 2350 but disregarded in computing an image valuefor point 2301. In some embodiments, rays to be disregarded are detectedby a mismatch with the color of the central ray. In some embodiments theresulting pixel value is normalized by dividing by the fraction of raysthat are not disregarded. These embodiments may be used in isolation orcombination with each other and any other embodiments, including thosedirected to extending the depth of field.

As discussed above, light data is processed in accordance with variousexample embodiments to focus and/or correct images. A variety ofapproaches to the latter correction approach are discussed as follows.In some of these embodiments, aberrations are corrected by tracing raysthrough the actual optical elements of the optics (e.g., lens or lenses)used in capturing the rays, and mapping the traced rays to particularphotosensors capturing the light. Light data is rearranged, using knowndefects exhibited by the optics as well as the known position of thesensors detecting the light.

In one correction-type embodiment, the world of rays that contribute toeach pixel as formed through idealized optics is computed for each pixelon a film of a synthesized photograph. In one implementation, these raysare computed by tracing rays from the virtual film location back throughthe ideal optics into the world. FIG. 15 illustrates an approach to thetracing of rays from a 3D point 1501 on a virtual film plane 1510through an ideal thin main lens 1520 out into a cone of world rays 1530,in connection with one such example embodiment. In some implementations,the set of desired rays 1525 may not necessarily correspond to directionthrough a real lens, but may correspond to any set of rays that are tobe weighted and summed to produce a desired image value.

FIG. 16 shows an approach to finding the value of light traveling alongideal rays for a particular application, in connection with anotherexample embodiment. These values are computed by tracing desired idealworld rays 1630 through a real main lens 1650, having a single elementwith spherical interfaces, used to physically direct the real worldlight rays to the ray sensor 1640 at the time the rays are measured(detected). In this embodiment, the rays that ideally converge to asingle 3D point (1601) do not converge, representing a defect of lenseswith spherical interfaces called spherical aberration. The ray sensor1640 provides individual values for each of the aberrated rays (such as1651), which are used to correct for the spherical aberration.

FIGS. 17A-17C illustrate example results using a computer simulationwith an approach to lens correction. The image in FIG. 17A is an ideal512×512 photograph (as seen through perfect optics). The image in FIG.17B is an image that would be produced with a real f/2 bi-convexspherical lens, which has loss in contrast and blurring. The image inFIG. 17C is a photograph computed using an approach to image correctiondescribed above, using an optics and sensor arrangement facilitating10×10 directional (u, v) resolution at each of 512×512 microlenses.

In another example embodiment of the present invention, chromaticaberrations are corrected in a main lens used to capture an image.Chromatic aberration is caused by the divergence of rays of light asthey are physically directed through optics because of differences inthe physical direction dependent on the wavelength of light. Theincoming rays are traced through the actual optics, taking into accountthe wavelength-dependent refraction of light that occurs in the actualoptics. In some applications, each color component of the system istraced separately based on the primary wavelength.

In another example embodiment, each of the red, green and bluecomponents common in color imaging systems is traced separately, asillustrated in FIG. 18A. The green world light rays are computationallytraced back into the imaging system to produce green rays 1830, and todetermine where they intersect a color ray sensor 1810 and at whatdirection they intersect the color ray sensor 1810. Similarly, FIG. 18Billustrates computationally tracing the desired blue world light rays1820, which are refracted to a greater extent than the green light rays.FIG. 18C illustrates computationally tracing the desired red world lightrays 1830, which are refracted to a lesser extent than the green lightrays. The values for each ray are computed from the values from the raysensor 1810 using, for example, approaches discussed in connection withother example embodiments described herein. The light field values foreach ray are integrated to calculate the corrected image value for eachparticular film pixel. For some applications, chromatic aberration isameliorated by refocusing each color channel on the plane at which itswavelengths best come into focus.

The desired light rays may not converge exactly on one of the discreteray values sampled by the ray sensor. In some embodiments, the value tobe used for such rays is computed as a function of the discrete rayvalues. In some embodiments, this function corresponds to a weighted sumof the value of discrete rays in a neighborhood of the desired lightray. In some implementations, this weighted sum corresponds to a 4Dconvolution of the discrete sample values with a predeterminedconvolution kernel function. In other implementations, the weighting maycorrespond to a quadrilinear interpolation from the 16 nearestneighbors. In still other implementations, the weighting may correspondto a cubic or bicubic interpolation from the 16 nearest neighbors.

It is worth noting that example correction processes have been describedin terms of ray-tracing for conceptual simplicity; a variety of otherapproaches are implemented with correction. In one embodiment, for eachdesired output pixel, the set of photosensor values that contribute arepre-computed along with their relative weights. As described above,these weights are a property of a number of factors that may include theoptics, sensor, desired set of rays to be weighted and summed for eachoutput pixel and desired light field reconstruction filter. Theseweights are pre-computed, selectively using ray-tracing, and stored. Acorrected image is formed by weighting and adding the appropriate sensedlight field values for each output pixel.

FIG. 29 and FIG. 30 illustrate other example embodiments for use inconnection with the above correction approach. FIG. 29 is a process flowdiagram for pre-computing a database of weights associated with ray(light) sensors and an output pixel value associated with each raysensor. In the first two blocks 2910 and 2920, a data set (e.g., in adatabase) is received for a desired image formation process consisting(for each output image pixel) of a set of ideal world light rays to besummed to produce an output image pixel value, and a specification forreal main lens optics used to physically direct light rays to a lightray sensor. At block 2925, an image pixel is chosen. For the outputvalue of this pixel, the associated set of world rays is computationallytraced at block 2930 through a virtual representation of the main lensoptics to the ray sensor. This results in a set of weights to be appliedto each ray sensor value to compute the output pixel value. These valuesare stored in an output database in block 2940. Block 2950 checkswhether all pixels have been processed, returning to block 2925 if not.If all pixels have been processed, the final block 2960 saves thecompleted database.

FIG. 30 is a flow diagram for a process that computes an output imageusing a database of weights that may have been computed by a process asin FIG. 29. In block 3010 and 3020, the process receives the databaseand a set of ray sensor values captured with the main lens optics usedin computing the database. At block 3025, a pixel in the output image isselected, so that its final image value may be computed. For theselected pixel, block 3030 uses the database to find the set of raysensor that contributes and their weights. In block 3040, each sensorvalue given in 3020 is weighted and added to a sum for that image pixelvalue. At block 3050, a check is performed to see whether all imagepixels have been processed. If not, the process returns to block 3025,and if so, the output image is saved at block 3060.

In a variety of example embodiments, light data is processed in thefrequency domain, with certain approaches directed to computationalapproaches to refocusing that operate in the Fourier domain. FIG. 20 isa flow diagram illustrating one such approach, in connection withanother example embodiment. The input to the algorithm is a discrete 4Dlight field 2010, which we will refer to as L(s,t,u,v), representing theray starting at (u, v) on a main lens and terminating at (s, t) on amicrolens plane (e.g., from the main lens 110 and terminating at a planeof the microlens array 120 of FIG. 1). The first step is to compute thediscrete 4D Fourier transform 2020 of the light field. The 4D Fouriertransform value at (k_(s), k_(t), k_(u), k_(v)), let us call this valueM(k_(s), k_(t), k_(u), k_(v)) is defined by the following equation:

M(k _(s) , k _(t) , k _(u) , k _(v))=∫∫∫∫L(s, t, u, v)exp(−π√{squareroot over (−1)}·(sk _(s) +tk _(t) +uk _(u) +vk _(v)))ds dt du dv,   (1)

where the exp function is the exponential function, exp(x)=e^(x). Insome embodiments the discrete light field is sampled on a rectilineargrid in the 4D space, and the Fourier transform is computed with theFast Fourier Transform (FFT) algorithm.

The next step, which is executed once for each depth at which we wish torefocus the image, is to extract appropriate 2D slices 2030 of the 4DFourier transform, and compute the inverse 2D Fourier transforms of theextracted slices, which are photographs focused at different depths2040. The inverse 2D Fourier transform, g(x, y), for a function G(k_(x),k_(y)) is defined by the following equation:

g(x, y)=∫∫G(k _(x) , k _(y))exp(2π√{square root over (−1)}·(xk _(x) +yk_(y)))dk _(x) dk _(y).   (2)

The values on the extracted 2D slice are determined by the depth atwhich we want to refocus. Considering the conjugate plane (on theimage-side of the lens) for the desired world focal plane, when theseparation between this conjugate plane and the main lens is D and theseparation between the microlens plane and the main lens is F, then thevalue of the extracted 2D slice at coordinates (k_(x),k_(y)) is given by

G(k _(x) ,k _(y))=1/F ² ·M(k _(x)(1D/F), k _(y)(1−D/F), k _(x) D/F, k_(y) D/F).   (3)

Using various approaches, artifacts that result from discretization,resampling and Fourier transformation are selectively ameliorated. Ingeneral signal-processing terms, when we sample a signal it isreplicated periodically in the dual domain. When we reconstruct thissampled signal with convolution, it is multiplied in the dual domain bythe Fourier transform of the convolution filter. In this regard, theoriginal, central replica is isolated, eliminating all other replicas. Adesirable filter is a 4D sinc function, sinc(s)sinc(t)sinc(u)sinc(v),where sinc(x)=sin(πx)/(πx); however, this function has infinite extent.

In various approaches, finite-extent filters are used withfrequency-domain processing; such filters may exhibit defects, which areselectively mitigated. FIGS. 21A illustrate these defects with respectto a specific 1D filter, with corresponding discussion below directed tomitigation or such defects. FIG. 21A represents a triangle filterapproach implemented with in linear interpolation in 1D (or as the basisfor a 4D quadrilinear filter). FIG. 21B shows the Fourier transform ofthe triangle filter approach, which is not of unit value within theband-limit (see 2010), and which gradually decays to smaller fractionalvalues as the frequency increases. In addition, the filter is not trulyband-limited, containing energy at frequencies outside the desiredstop-band (2020).

The first defect described above leads to “rolloff artifacts,” which canlead to a darkening of the borders of computed photographs. Decay in thefilter's frequency spectrum with increasing frequency means that thespatial light field values, which are modulated by this spectrum, also“roll off' to fractional values towards the edges.

The second defect described above involves aliasing artifacts incomputed photographs, which are related to energy at frequencies abovethe band-limit. The non-zero energy that extends beyond the band-limitmeans that the periodic replicas are not fully eliminated, leading totwo kinds of aliasing. First, the replicas that appear parallel to theslicing plane appear as 2D replicas of the image encroaching on theborders of the final photograph. Second, the replicas positionedperpendicular to this plane are projected and summed onto the imageplane, creating ghosting and loss of contrast.

In an example embodiment, correction for rolloff-type defects asdescribed above is eliminated by multiplying the input light field bythe reciprocal of the filter's inverse Fourier spectrum, to nullify theeffect introduced during resampling. In this example embodiment,multiplication is performed prior to taking the 4D Fourier transform inthe pre-processing step of the algorithm. While it corrects rollofferror, pre-multiplication may accentuate the energy of the light fieldnear its borders, maximizing the energy that folds back into the desiredfield of view as aliasing.

Three methods of suppressing aliasing artifacts—oversampling, superiorfiltering and zero-padding—are used individually or in combination invarious example embodiments described below. Oversampling within theextracted 2D slice increases the replication period in the spatialdomain. This means that less energy in the tails of the in-planereplicas will fall within the borders of the final photograph.Increasing the sampling rate in one domain leads to an increase in thefield of view in the other domain. Aliasing energy from neighboringreplicas falls into these outer regions, which is cropped away toisolate the original, central image of interest.

Another approach to mitigating aliasing is directed to a finite-extentfilter that approximates a perfect spectrum (as would be exhibited viause of an ideal filter) as closely as possible. In an exampleembodiment, a 4D Kaiser-Bessel separable function,kb4(s,t,u,v)=kb(s)kb(t)kb(u)kb(v), is used as the filter, where

kb(x)=1/W·I ₀(P·√{square root over (1−(2x/W)²)})   (4)

In this equation, I₀ is the standard zero-order modified Kaiser-Besselfunction of the first kind, W is the width of the desired filter, and Pis a parameter that depends on W.

In this example embodiment, W values are 5, 4.5, 4.0, 3.5, 3.0, 2.5, 2.0and 1.5, and the P values are, respectively, 7.4302, 6.6291, 5.7567,4.9107, 4.2054, 3.3800, 2.3934, and 1.9980. For general informationregarding aliasing, and for specific information regarding approaches tomitigating aliasing in connection with one or more example embodimentsof the present invention, reference may be made to Jackson J. I., MeyerC. H., Nishimura, D. G. and Macovski, A., 1997, Selection of convolutionfunction for Fourier inversion using gridding. IEEE Transactions onMedical Imaging, 10, 3, 473-478, which is fully incorporated herein byreference. In one implementation, widths “W” of less than about 2.5 areimplemented to achieve desirable image quality.

In another example embodiment, aliasing is mitigated by padding a lightfield with a small border of zero values before pre-multiplication andtaking its Fourier transform. This pushes energy slightly further fromthe borders, and minimizes the amplification of aliasing energy by thepre-multiplication for rolloff correction.

FIG. 22 is a flow diagram showing an approach to refocusing in thefrequency domain using various corrections described above, according toanother example embodiment of the present invention. At block 2210, adiscrete 4D light field is received. In the pre-processing phase, whichoccurs once per input light field, block 2215 checks if aliasingreduction is desired, and if so executes block 2220, which pads thelight field with a small border (e.g., 5% of the width in thatdimension) of zero values. At block 2225 a check is performed todetermine whether rolloff correction is desired, and if so, the lightfield is modulated at block 2230 by the reciprocal of the Fouriertransform of the resampling filter. In the final block of thepre-processing phase, the 4D Fourier transform of the light field iscomputed at block 2240.

In the refocusing phase, which occurs once per desired focal depth, theprocess receives a desired focal depth of refocused image at block 2250,such as through the direction of a user. At block 2260, a check isperformed to determine whether aliasing reduction is desired. If not,block 2270 extracts a 2D slice of the Fourier transform of the lightfield, with a desired 4D r sampling filter, where the trajectory of the2D slice corresponds to the desired focal depth; and block 2275 computesan inverse 2D Fourier transform of the extracted slice and moves toblock 2290. If aliasing reduction was desired at block 2260, the processmoves to block 2280, at which a 2D slice with desired 4D resamplingfilter and oversampling (e.g. 2× oversampling in each of the twodimensions) is extracted. At block 2283, the slice's inverse 2D Fouriertransform is computed, and the resulting image is cropped to theoriginal size without oversampling at block 2286, after which theprocess moves to block 2290. At block 2290, a check is performed todetermine whether refocusing is complete. If not, another focal depth ischosen at block 2250 and the process proceeds as described above. Ifrefocusing is complete, the process exits at block 2295.

The asymptotic computational complexity of this frequency-domainalgorithm is less than refocusing by explicitly summing rays asdescribed for the alternate embodiment above. Assume that the inputdiscrete light field has N samples in each of its four dimensions. Thenthe computational complexity of the algorithm that explicitly sums raysis O(N⁴) for refocusing at each new depth. The computational complexityof the frequency-domain algorithm is O(N² log N) for refocusing at eachnew depth, dominated by the cost of the inverse 2D Fourier transform.However, the pre-processing step costs O(N⁴ log N) for each new lightfield dataset.

In another example embodiment, the captured light rays are opticallyfiltered. Although not limited to such applications, some examples ofsuch filters are neutral density filters, color filters, polarizingfilters. Any filter currently existing or that may be developed in thefuture may be used to effect a desired filtering of the rays of light.In one implementation, the light rays are optically filtered in groupsor individually, so that each group or individual ray is filtereddifferently. In another implementation, a filtering is applied by theuse of a spatially-varying filter attached to a main lens. In oneexample application, a gradient filter such as a neutral-densitygradient filter is used to filter light. In another implementation,spatially varying filters are used in front of one or more of a raysensor, a microlens array or a photosensor array. Referring to FIG. 1 byway of example, one or more such filters are selectively placed in frontof one or more of the main lens 110, microlens array 120 and photosensorarray 130.

In another example embodiment of the present invention, a computationalcomponent such as a processor is programmed to selectively choose raysto combine in computing output pixels in order to effect a desired netfiltering for that pixel value. By way of example, consider embodimentsinvolving an optical neutral gradient density filter at the main lens,each image of the lens aperture that appears under a microlens isweighted by the filter gradient across its extent. In oneimplementation, output images are computed by selecting a photosensorunder each microlens at the point of the gradient that matches thedesired level of neutral-density filtering for that output image pixel.For example, to produce an image in which every pixel is filtered to alarge extent, every pixel value is set to the value of the photosensorunder the corresponding microlens that is at the extreme end of thegradient corresponding to maximum filtering.

FIG. 2 is a data-flow diagram showing an approach to processing imagesin connection with other example embodiments of the present invention.An image sensor arrangement 210 collects image data usingmicrolens/photosensor chip arrangement 212 in a manner similar, forexample, to the microlens array 120 and photosensor array 130 shown inFIG. 1 and described above. The image sensor arrangement 210 optionallyincludes an integrated processing circuit 214 bearing certain processingcircuitry to prepare collected image data for transfer.

Sensor data created at the image sensor arrangement 210 is passed to asignal processor 220. The signal processor includes a low-resolutionimage processor 222 and one or both of a compression processor 224 and a(light) ray-direction processor 226; each of these processors isselectively implemented separately or functionally with a commonprocessor, depending upon the application. Furthermore, each of theprocessors shown in FIG. 2 is selectively programmed with one or moreprocessing functions described in connection with other figures orelsewhere herein. The signal processor 220 is optionally implemented ina common device or component with the image sensor arrangement 210, suchas on a common circuit and/or in a common image device.

The low-resolution image processor 222 uses sensor data received fromthe image sensor arrangement 210 to generate low-resolution image data,which is sent to a viewfinder display 230. An input device 235, such asa pushbutton on a camera or video camera, sends an image capture requestto the signal processor 220 requesting, for example, the capture of aparticular image displayed in the viewfinder display 230 and/or toinitiate video imaging where so implemented.

In response to the image capture request or as otherwise directed, thesignal processor 220 uses the sensor data captured by the image sensorarrangement 210 to generate processed sensor data. In some applications,the compression processor 224 is implemented to generate compressed rawdata for transfer to a data storage arrangement 240 (e.g., memory). Suchraw data is then selectively processed at the signal processor 220and/or at an external computer 260 or other processing device,implementing ray-direction processing such as that implemented with theray-direction processor 226, which is discussed further below.

In certain applications, the ray-direction processor 226 is implementedto process the sensor data received at the signal processor 220 torearrange the sensor data for use in generating focused and/or correctedimage data. The ray-direction processor 226 uses one or both of sensordata received from the image sensor arrangement 210 and raw data sent tothe data storage arrangement 240. In these applications, theray-direction processor 226 uses ray-mapping characteristics of theparticular imaging device (e.g., camera, video camera or mobiletelephone) in which the image sensor arrangement 210 is implemented todetermine a rearrangement of light rays sensed with themicrolens/photosensor chip 212. Image data created with theray-direction processor 226 is sent to the data storage arrangement 240and/or to a communication link 250 for use in a variety of applications,such as in streaming image data or otherwise sending image data to aremote location.

In some applications, the integrated processing circuit 214 includessome or all of the processing functionality of the signal processor 220by implementing, for example, a CMOS-type processor or other processorwith appropriate functionality. For instance, the low-resolution imageprocessor 222 is selectively included with the integrated processingcircuit 214, with the low-resolution image data sent directly to theviewfinder display 230 from the image sensor arrangement 210. Similarly,the compression processor 224, or functionality similar thereto, isselectively implemented with the integrated processing circuit 214.

In some applications, computation of final images may be performed onthe integrated processing circuit 214 (e.g. in some digital stillcameras that output only final images). In other applications, the imagesensor arrangement 210 may simply transmit the raw light ray data, or acompressed version of these data, to an external computational device,such as a desktop computer. Computation of final images from these datais then performed on the external device.

FIG. 3 is a flow diagram for a method for processing image data,according to another example embodiment of the present invention. Atblock 310, image data is captured at a camera or other imaging device,using a main lens or a lens stack with a microlens/photosensor arraysuch as that shown in FIG. 1. If a preview image is desired at block320, the preview image is generated at block 330 using, for example, aviewfinder or other type of display. The preview image is displayed, forexample, in a viewfinder of a camera or video camera, using a subset ofthe captured image data.

Raw data from the photosensor array is processed and compressed for useat block 340. Light ray data is extracted from the processed andcompressed data at block 350. This extraction involves, for example,detecting a bundle or set of light rays incident upon a particularphotosensor in the photosensor array. Ray mapping data is retrieved atblock 360 for the imaging arrangement in which the image data iscaptured. The ray-mapping data and the extracted light ray data is usedto synthesize a re-sorted image at block 370. For example, theextraction, mapping and synthesis blocks 350-370 are selectivelyimplemented by determining a bundle of light rays for a particular pixelof a scene for which the light rays were collected, and integrating theenergy of the light rays to synthesize a value for the particular pixel.In some applications, the ray mapping data is used to trace light raysfor each particular pixel through actual lenses used to acquire theimage data. For example, by determining an appropriate set of rays toadd together in order to focus upon a selected subject at a particularfocal depth at block 370, the rays can be re-sorted to arrive at afocused image. Similarly, by determining a proper arrangement of rays tocorrect for conditions such as lens aberrations in the imaging device,the rays can be re-sorted to generate an image relatively free ofcharacteristics relating to the aberrations or other condition.

A variety of approaches are selectively used to generate preview imagesfor camera-type and other applications. FIG. 4 is a process flow diagramfor generating such a preview image, according to another exampleembodiment of the present invention. The approach shown in FIG. 4 anddiscussed below may be implemented, for example, in connection with thegeneration of a preview image at block 330 of FIG. 3.

A preview instruction with raw sensor image data is received at block410. At block 420, a center pixel is selected from each microlens imagein the raw sensor image data. The selected center pixels are collectedto form a high depth-of-field image at block 430. At block 440, the highdepth-of-field image is downsampled to a resolution amenable for use ina viewfinder display. Referring to FIG. 2 by way of example, suchdownsampling is selectively performed at one or more of the image sensorarrangement 210 or the signal processor 220. The generated preview imagedata is sent to the viewfinder display at block 450, and at block 460,the viewfinder displays an image with the preview image data.

FIG. 5 is a process flow diagram for processing and compressing imagedata, according to another example embodiment of the present invention.The approach shown in FIG. 5 and discussed below may be implemented, forexample, in connection with the processing and compressing of image dataat block 340 of FIG. 3. When implemented with an arrangement as shown inFIG. 2, the approach shown in FIG. 5 may, for example, be implemented atone or both of the image sensor arrangement 210 and the signal processor220.

At block 510, raw image data is received from a sensor array. Ifcoloring is desired at block 520, color filter array values aredemosiaced at block 530 to produce color at the sensors. Ifrectification and alignment is desired at block 540, microlens imagesare rectified and aligned with the photosensor array at block 550. Ifinterpolation is desired at block 560, pixel values are interpolated atblock 570 to an integral number of pixels associated with eachmicrolens. At block 580, the processed raw image data is compressed andpresented for synthesis processing (e.g., to form a refocused and/orcorrected image).

FIG. 6 is a process flow diagram for image synthesis, according toanother example embodiment of the present invention. The approach shownin FIG. 6 and discussed below may be implemented, for example, inconnection with the image synthesis approach shown at block 370 of FIG.3 and discussed further below.

At block 610, raw image data is received from a photosensor array. Ifrefocusing is desired at block 620, image data is refocused at block 630using, e.g., approaches discussed herein for selectively re-sortinglight represented by the raw image data. If image correction is desiredat block 640, image data is corrected at block 650. In variousapplications, image correction at block 650 is carried out before orconcurrently with refocusing at block 630 in applications where bothrefocusing and image correction is desirable. A resultant image isgenerated at block 660 using processed image data including refocusedand corrected data, where applicable.

FIG. 7A is a process flow diagram for image refocusing with a lensarrangement, according to another example embodiment of the presentinvention. The approach shown in FIG. 7 and discussed below may beimplemented, for example, in connection with the refocusing of imagedata at block 630 in FIG. 6.

At block 710, a virtual focal plane for refocusing an image portion isselected. At block 720, a pixel of a virtual image for the virtual focalplane is selected. If correction (e.g., for lens aberration) is desiredat block 730, the value of a virtual light ray (or virtual set of lightrays) passing between the selected pixel and each particular lensposition is calculated at block 740. In one application, thiscalculation is facilitated by computing the conjugate light ray thatwould fall upon the selected pixel and tracing that ray through the pathin the lens arrangement.

At block 750, the sum of light ray (or virtual set of light ray) valuesfor each lens position for the particular focal plane are added todetermine a total value for the selected pixel. In some applications,the sum added at block 750 is a weighted sum, wherein certain light rays(or set of light rays) are given greater weight than others. If thereare additional pixels for refocusing at block 760, another pixel isselected at block 720 and the process continues until no further pixelsare desirably refocused. After the pixels have been refocused, the pixeldata is combined at block 770 to generate a refocused virtual image atthe virtual focal plane selected in block 710. The refocusing approachesinvolving some or all of blocks 720, 730, 740 and 750 in FIG. 7 arecarried out via more specific functions for a variety of applications.

The sensor data processing circuitry implemented with one or moreexample embodiments described herein includes one or moremicroprocessors, Application-Specific Integrated Circuits (ASICs),digital signal processors (DSPs), and/or programmable gate arrays (forexample, field-programmable gate arrays (FPGAs)), depending upon theimplementation. In this regard, sensor data processing circuitry may beany type or form of circuitry whether now known or later developed. Forexample, the sensor data processing circuitry may include a singlecomponent or a multiplicity of components (microprocessors, ASICs andDSPs), either active and/or passive, which are coupled together toimplement, provide and/or perform a desiredoperation/function/application.

In various applications, the sensor data processing circuitry performsor executes one or more applications, routines, programs and/or datastructures that implement particular methods, tasks or operationsdescribed and/or illustrated herein. The functionality of theapplications, routines or programs are selectively combined ordistributed in certain applications. In some applications, theapplications, routines or programs are implemented by sensor (or other)data processing circuitry using one or more of a variety of programminglanguages, whether now known or later developed. Such programminglanguages include, for example, FORTRAN, C, C++, Java and BASIC, whethercompiled or uncompiled code, selectively implemented in connection withone or more aspects of the present invention.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the invention.Based on the above discussion and illustrations, those skilled in theart will readily recognize that various modifications and changes may bemade to the present invention without strictly following the exemplaryembodiments and applications illustrated and described herein. Forinstance, such changes may include implementing the various opticalimaging applications and devices in different types of applications,increasing or decreasing the number of rays collected per pixel (orother selected image area), or implementing different algorithms and/orequations than the examples described to assemble or otherwise processimage data. Other changes may involve using coordinate representationsother than or in addition to Cartesian coordinates, such as polarcoordinates. Such modifications and changes do not depart from the truespirit and scope of the present invention.

1-21. (canceled)
 22. A digital imaging system for generating a virtualimage of a scene, wherein the virtual image includes a plurality ofpixels, the system comprising: a photosensor array, having a pluralityof photosensors, configured to detect light including a plurality oflight rays from the scene, wherein each photosensor, in response todetecting the light incident thereon, generates light data, thephotosensor array further configured to detect a set of the plurality oflight rays that concurrently arrive at the particular portion of thephysical focal plane at different angles of incidence; an opticsarrangement, having a physical focus, configured to direct light raysfrom the scene to the photosensor array via a physical focal plane; andprocessing circuitry to compute a pixel value of each pixel of thevirtual image by combining light data from selected photosensors basedon angles of incidence of the light rays detected by each photosensor,as characterized by the location of each photosensor relative to thephysical focal plane, the virtual image having a virtual focus and avirtual focal plane that are respectively different from the physicalfocus and the physical focal plane, wherein the virtual focal planecomprises a planar focal plane and a non-planar focal plane, wherein theprocessing circuitry further determines the angles of incidence of lightupon each photosensor using a position of each photosensor relative tothe microlens.
 23. The system of claim 22, wherein the processingcircuitry is configured to generate a virtual image of the scene bycombining selected light rays based on aberrations in the opticsarrangement.
 24. The system of claim 22, wherein the processingcircuitry is configured to determine an angle of incidence of thedetected light rays as characterized by the location of each photosensordetecting the light, determine aberrated rays using the angle ofincidence of the detected light rays, re-sort the aberrated rays tocorrect for aberrations in the optics arrangement, and generate thevirtual image of the scene by combining selected light rays and resortedaberrated rays.
 25. The system of claim 22, wherein the processingcircuitry is further configured to determine an angle of incidence ofthe detected light rays as characterized by the location of eachphotosensor detecting the light, map the path of the light rays usingthe angle of incidence of the detected light rays and tracing of raysthrough the optics arrangement, determine aberrated rays using themapped light rays, re-sort the aberrated rays, and generate a virtualimage of the scene by combining selected light rays and re-sortedaberrated rays to reduce the effect of aberrations in the opticsarrangement.
 26. A digital imaging system for generating a virtual imageof a scene, the system comprising: optics, having a physical focus, todirect light from the scene onto a physical focal plane, wherein thelight includes a set of light rays; a photosensor array, having aplurality of photosensors, to detect the set of light rays, wherein eachphotosensor detects different ones of the set of light rays; a microlensarray, disposed between the optics and the photosensor array, andconfigured to physically direct the set of light rays from the optics tothe photosensor array; and processing circuitry configured to generate avirtual image of the scene using the detected set of light rays byselectively combining data representing the light rays based ondirectional characteristics of the light rays, wherein the virtual imageincludes a virtual focus at a virtual focal plane, wherein the virtualfocus and the virtual focal plane are different from the physical focusand the physical focal plane, respectively.
 27. The digital imagingsystem of claim 26, wherein the processing circuitry is furtherconfigured to combine selected light rays based on aberrations in theoptic arrangement.
 28. A method of imaging a scene using a digitalimaging device which includes an optics unit and a light ray sensor, themethod comprising: focusing light from the scene upon a physical focalplane; detecting the light from the scene which is incident on thephysical focal plane; determining directional characteristics of thedetected light; determining light rays corresponding to the detectedlight using the directional characteristics of the detected light; andgenerating a virtual image of the scene having a virtual focus at avirtual focal plane using the detected light by combining selected lightrays based on the directional characteristics of the light, wherein thevirtual focus and the virtual focal plane are different from thephysical focus and physical focal plane, respectively.
 29. The method ofclaim 28 wherein: generating the virtual image of the scene by combiningselected light rays to provide the virtual image of the scene having avirtual focus at a virtual focal plane further includes generating thevirtual image of the scene by combining selected light rays based onaberrations in the optics unit and/or light ray sensor.
 30. The methodof claim 28 further including: determining aberrated rays using thedetermined light rays corresponding to the detected light and thedirectional characteristics of the light arriving at different locationson the physical focal plane; re-sorting the aberrated rays to correctfor aberrations in one or more of the optics unit and light ray sensor;and wherein generating the virtual image of the scene further includescombining selected light rays and re-sorted aberrated rays.
 31. Themethod of claim 28 further including: mapping the light rayscorresponding to the light using the directional characteristics of thelight and tracing of rays through one or more of the optics unit andlight ray sensor; determining aberrated rays using the mapped lightrays; re-sorting the aberrated rays; and wherein generating the virtualimage of the scene further includes combining selected light rays andre-sorted aberrated rays to reduce the affect of aberrations in theoptics unit and/or light ray sensor.